Skip to content

feat: add metadata hash checks #924

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jan 13, 2025
Merged

feat: add metadata hash checks #924

merged 11 commits into from
Jan 13, 2025

Conversation

ntn-x2
Copy link
Member

@ntn-x2 ntn-x2 commented Dec 20, 2024

Not a breaking change, users can opt in based on the new feature PR introduced in KILTprotocol/kilt-node#835. I did not write any tests yet, but I have modified some and run them against a local instance with the new feature, and everything works flawlessly.

Checklist

  • Fix yarn.lock issue
  • Fix latest-develop Docker image issue
  • Write basic integration tests against the latest-develop image for metadata hash checks

@ntn-x2 ntn-x2 self-assigned this Dec 20, 2024
@ntn-x2 ntn-x2 requested review from rflechtner and Dudleyneedham and removed request for rflechtner December 20, 2024 14:06
@ntn-x2 ntn-x2 assigned ntn-x2, rflechtner and Dudleyneedham and unassigned ntn-x2 Dec 20, 2024
@ntn-x2 ntn-x2 requested a review from rflechtner December 20, 2024 14:07
ntn-x2 added a commit to KILTprotocol/kilt-node that referenced this pull request Jan 6, 2025
We introduce a new feature for peregrine and spiritnet runtimes, which
adds the metadata hash signed extension, for wallets that need it.

The Gitlab build pipeline has also been updated to include the new
feature only when building the production WASMs.

## How to test

Change any integration tests in the SDK based on this PR
(KILTprotocol/sdk-js#924), and verify the
metadata is verified (e.g., by enabling logs via a Chopsticks
deployment).
@ntn-x2
Copy link
Member Author

ntn-x2 commented Jan 6, 2025

@rflechtner I am not understanding what I am doing wrong. I have inconsistent results running yarn install and yarn build on my machine and in the CI.

@Dudleyneedham
Copy link
Member

Check your node version. Also, check what version of Yarn you are running compared to the package.json

@ntn-x2
Copy link
Member Author

ntn-x2 commented Jan 6, 2025

@rflechtner after changing the dependency type from peer dependency back to regular dependency, everything seems to be building ok and tests seem to pass (minor what was already broken on develop). I would then ask you to provide your feedback, or if you think the dependency should be a peer dependency, to try to fix it yourself as I honestly have no clue.

@rflechtner
Copy link
Contributor

@rflechtner after changing the dependency type from peer dependency back to regular dependency, everything seems to be building ok and tests seem to pass (minor what was already broken on develop). I would then ask you to provide your feedback, or if you think the dependency should be a peer dependency, to try to fix it yourself as I honestly have no clue.

Peer dependencies have no effect besides issuing warnings in the CLI, so my guess the dependency was not actually being installed. I'm fine with leaving it as a regular dependency, given that it's from an entirely different stack of dependencies than the @polkadot/api related code. On this note though, is there no support from the @polkadot/api suite for checking the metadata hash? I'm not sure it's a great idea to start mixing the pjs and papi toolchains.

@ntn-x2
Copy link
Member Author

ntn-x2 commented Jan 7, 2025

@rflechtner gotcha 👌

As for the library, unfortunately pjs itself does not offer the feature. The available options are the ones listed in this Polkadot Forum. Even the Polkadot Apps code relies on the @polkadot/api library for this one feature: https://github.com/polkadot-js/apps/blob/f63da86b9a29bc9a99aaa18c75db61c242b9d9fd/packages/react-signer/src/signers/LedgerSigner.ts#L15.

@ntn-x2
Copy link
Member Author

ntn-x2 commented Jan 7, 2025

Anyway, in order to test this, we first must fix an issue with our build pipeline that is failing to push new Docker images on DockerHub (just found out). @ggera is on it. Once that's resolved, I will try to push an empty commit to re-trigger the tests, and see if everything works fine. This should also fix the broken pipeline currently on develop, since the fixes for the missing runtime APIs have not been picked up cause there was no image pushed.

@rflechtner
Copy link
Contributor

@rflechtner gotcha 👌

As for the library, unfortunately pjs itself does not offer the feature. The available options are the ones listed in this Polkadot Forum. Even the Polkadot Apps code relies on the @polkadot/api library for this one feature: https://github.com/polkadot-js/apps/blob/f63da86b9a29bc9a99aaa18c75db61c242b9d9fd/packages/react-signer/src/signers/LedgerSigner.ts#L15.

I see! https://www.npmjs.com/package/merkleized-metadata seems lighter on additional dependencies, but choose whichever you think gets the job done.

@rflechtner
Copy link
Contributor

Once that's resolved, I will try to push an empty commit to re-trigger the tests, and see if everything works fine.

You can also just hit the 'Re-run jobs' button on the actions status page (https://github.com/KILTprotocol/sdk-js/actions/runs/12633893068?pr=924)

@Dudleyneedham
Copy link
Member

@rflechtner The only concern with this package it doesn't seem maintained. I offered this yesterday but I think we should find another package if possible that is maintained or stick with Papi.

@ntn-x2
Copy link
Member Author

ntn-x2 commented Jan 7, 2025

I think re-running the same job does not pull the latest version of a Docker image, does it? I think I tried this once, but sure I will try first to re-run the job once the new images are available.

@ntn-x2
Copy link
Member Author

ntn-x2 commented Jan 7, 2025

@rflechtner The only concern with this package it doesn't seem maintained. I offered this yesterday but I think we should find another package if possible that is maintained or stick with Papi.

+1 on this. I think we should stick with papi.

@ntn-x2
Copy link
Member Author

ntn-x2 commented Jan 7, 2025

@rflechtner resetting to develop and re-installing dependencies still changes all hashes. So I force-pushed changes to the files and package.json, but not yarn.lock. Would you be able to fix that, and then I could go ahead with writing some integration tests for it?

@ntn-x2 ntn-x2 marked this pull request as draft January 7, 2025 11:00
@ntn-x2
Copy link
Member Author

ntn-x2 commented Jan 8, 2025

After spending a lot of time on this and not understanding why tests would pass when using Chopsticks but not when using the standalone-node binary, I came across this notice that says that the metadata check extension does not work with native runtimes. At the moment, we still use the NativeElseWasmExecutor for both our binaries, which means this feature is untestable until we switch to a pure WasmExecutor. Actually, the feature is unusable in general until we switch to the WasmExecutor, but until this PR is merged, nothing breaks as the default behaviour is to ignore the metadata hash.

@ntn-x2 ntn-x2 added the ✋on hold status: on hold label Jan 8, 2025
@ntn-x2 ntn-x2 removed the ✋on hold status: on hold label Jan 13, 2025
@ntn-x2 ntn-x2 marked this pull request as ready for review January 13, 2025 07:53
@ntn-x2 ntn-x2 requested a review from rflechtner January 13, 2025 07:53
@ntn-x2
Copy link
Member Author

ntn-x2 commented Jan 13, 2025

@rflechtner the CI is still broken due to missing runtime API decorations, but you can verify the changes work by running the new Blockchain integration tests locally.

Copy link
Contributor

@rflechtner rflechtner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have yet to test it, but the code looks great ! Just two little things you could adjust

@ntn-x2 ntn-x2 requested a review from rflechtner January 13, 2025 10:23
Copy link
Contributor

@rflechtner rflechtner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

Co-authored-by: Raphael Flechtner <39338561+rflechtner@users.noreply.github.com>
@ntn-x2 ntn-x2 merged commit 6eb7def into develop Jan 13, 2025
10 of 14 checks passed
@ntn-x2 ntn-x2 deleted the aa/metadata-hash branch January 13, 2025 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants